Agenda¶
- About Me
- A Short Glimpse of a Data Science Project
- Data Wrangling with Python
- Data Visualization with Python
A Short Glimpse of a Data Science Project¶
TL; DR¶
Data science is a huge field, it invovles applications and tools like importing, tidying, transformation, visualization, modeling, and communication. Surrounding all these is programming.
Source: R for Data Science
PEP 206(Python Enhancement Proposals 206): Python Advanced Library plays the key role¶
The Python source distribution has long maintained the philosophy of "batteries included": having a rich and versatile standard library which is immediately available¶
randomdatetimejson- ...etc.
However, the standard library modules aren't always the best choices for a specific application¶
A list of third-party modules that make Python the go-to choice for most of the data science teams¶
requestsnumpypandasmatplotlibscikit-learnscipytensorflowpytorchflaskdash- ...etc.
Import: This typically means that we take data stored in a file, database, or web API, and load it into a data structure in Python¶
csvjsonnumpypandasrequestsbeautifulsoup4lxmlsqlite3
Tidy: Storing data in a consistent form that matches the semantics of the dataset. Tidy data is important because the more consistent the easier for us to get it into the right form for different functions¶
numpypandas- Functional programming techniques
Transform: For instance narrowing in on observations of interest, creating new variables that are functions of existing variables, and calculating a set of summary statistics¶
numpypandas- Functional programming techniques
Visualize: A fundamental human activity. A good visualisation will show you things that you did not expect, or raise new questions about the data.¶
pandasmatplotlibseaborn
Model: Once we've made questions sufficiently precise, we can use a mathematical or computational tool to answer them¶
scikit-learnscipytensorflowpytorch
Communicate: It doesn’t matter how well our project is unless we can communicate the results to others¶
jupyterdash
Communication matters¶
It should be possible to explain the laws of physics to a barmaid.
Source: Einstein: His Life and Times
Data Wrangling with Python¶
TL; DR¶
We brief the what and why on pandas and a quickstart of common DataFrame operations with 91APP dataset.
What is pandas¶
Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more.
Why pandas¶
Python has always been a popular language for web scraping, however there is a weak spot in its analysis capability. Python used to switch to a more data-centric language like R or Matlab during the analysis stage.
Turns out it has nothing to do with panda...¶
pandas refers to three primary class customed by its author Wes McKinney¶
- Panel
- DataFrame
- Series
Data analysts mainly deal with Index, ndarray, Series, DataFrame¶
DataFrameSeriesIndexndarray
Libraries we need for the following demo¶
In [1]:
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import sqlite3
Combining an Index and a ndarray can form a Series¶
In [2]:
movie_index = pd.Index(['The Avengers', 'Avengers: Age of Ultron', 'Avengers: Infinity War', 'Avengers: Endgame'])
movie_ratings = np.array([8, 7.3, 8.5, 8.4])
print(type(movie_index))
print(type(movie_ratings))
In [3]:
rating_series = pd.Series(movie_ratings, movie_index)
print(type(rating_series))
print(rating_series)
And vice versa...¶
In [4]:
rating_series = pd.Series([8, 7.3, 8.5, 8.4], ['The Avengers', 'Avengers: Age of Ultron', 'Avengers: Infinity War', 'Avengers: Endgame'])
print(type(rating_series))
print(type(rating_series.index))
print(type(rating_series.values))
Combining multiple Series with the same Index can form a DataFrame¶
In [5]:
release_year_series = pd.Series([2012, 2015, 2018, 2019], ['The Avengers', 'Avengers: Age of Ultron', 'Avengers: Infinity War', 'Avengers: Endgame'])
movie_df = pd.DataFrame()
movie_df['rating'] = rating_series
movie_df['release_year'] = release_year_series
print(type(movie_df))
movie_df
Out[5]:
And vice versa...¶
In [6]:
print(type(movie_df['rating']))
print(type(movie_df['rating'].index))
Main pandas functionalities which data analysts rely on are¶
- Import
- Wrangling(Tidy + Transform)
- Visualize
pandas has massive functions importing tabular data¶
- Flat text file
- Database table
- Spreadsheet
- Array of JSONs
- ...etc.
Source: https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html
Flat text files: 91APP Dataset¶
- BehaviorDataForNTU.csv
- MemberDataForNTU.csv
- OrderDataForNTU.csv
- OrderSlaveDataForNTU.txt
Source: NTU資料分析與決策社
In [7]:
!file --mime-encoding *.csv
!file --mime-encoding *.txt
Importing using pandas read_csv functions¶
In [8]:
# for example
order_slave = pd.read_csv('OrderSlaveDataForNTU.txt')
In [9]:
order_slave.shape
Out[9]:
In [10]:
order_slave.dtypes
Out[10]:
Assigning data types if there is a warning message¶
In [11]:
dtypes = {
'PaymentType': 'str',
'ShippingType': 'str'
}
order_slave = pd.read_csv('OrderSlaveDataForNTU.txt', dtype=dtypes)
Why DtypeWarning?¶
Pandas is built on numpy, numpy records missing values with np.NaN which is a float type.
In [12]:
print(type(np.NaN))
Importing other files as well¶
In [13]:
dtypes = {
'CategoryId': 'str',
'SearchKeyWord': 'str',
'TransactionNum': 'str'
}
behavior = pd.read_csv('BehaviorDataForNTU.csv', dtype=dtypes)
member = pd.read_csv('MemberDataForNTU.csv')
order_main = pd.read_csv('OrderDataForNTU.csv')
Using shape attribute to check layout¶
In [14]:
print(behavior.shape)
print(member.shape)
print(order_main.shape)
print(order_slave.shape)
Using head method for a peek¶
In [15]:
behavior.head()
Out[15]:
In [16]:
member.head()
Out[16]:
In [17]:
order_main.head()
Out[17]:
In [18]:
order_slave.head()
Out[18]:
It is also recommended storing data in a database, especially for large dataset like BehaviorDataForNTU.csv¶
In [19]:
conn = sqlite3.connect('ntu_dac_91APP.db')
query_str = """
SELECT *
FROM order_slave
LIMIT 5;
"""
pd.read_sql(query_str, conn)
Out[19]:
Once we've done importing dataset, there are some common attributes/methods to examine a DataFrame object¶
shapeinfo()head()/tail()describe()
In [20]:
order_main.info()
In [21]:
order_main.describe()
Out[21]:
Once we've done importing and examination, we can move on to general DataFrame operations¶
- Selecting
- Filtering
- Subsetting
- Indexing
- Sorting
- Deriving
- Summarizing
- Summarizing and Grouping
Selecting columns from DataFrame¶
In [22]:
cols_to_select = ['RegisterDate', 'Gender', 'Birthday']
member[cols_to_select]
Out[22]:
Filtering rows from DataFrame¶
In [23]:
rows_to_select = member['Gender'] == 'Male'
member[rows_to_select]
Out[23]:
Subsetting columns and rows simultaneously from DataFrame¶
In [24]:
cols_to_select = ['RegisterDate', 'Gender', 'Birthday']
rows_to_select = member['Gender'] == 'Male'
member[rows_to_select][cols_to_select]
Out[24]:
Indexing DataFrame¶
loc[]: indexing withIndexiloc[]: indexing with absolute positions
In [25]:
male_members = member[rows_to_select]
male_members.loc[[53, 108, 109, 117, 120]]
Out[25]:
In [26]:
male_members = member[rows_to_select]
male_members.iloc[[0, 1, 2, 3, 4]]
Out[26]:
Sorting DataFrame¶
sort_valuesmethod: sorting with a specific columnsort_indexmethod: sorting with theIndexofDataFrame
In [27]:
order_main.sort_values('TotalSalesAmount')
Out[27]:
In [28]:
male_members.sort_index(ascending=False)
Out[28]:
Deriving new variables from DataFrame¶
mapwith adictmapwith a function(or a lambda expression)
In [29]:
member['RegisterSourceTypeDef'].unique()
Out[29]:
In [30]:
register_source_dict = {
'Store': 'Non mobile app',
'LocationWizard': 'Non mobile app',
'Web': 'Non mobile app',
'AndroidApp': 'Mobile app',
'iOSApp': 'Mobile app',
np.NaN: 'Non mobile app'
}
member['RegisterSourceTypeDef'].map(register_source_dict).value_counts()
Out[30]:
In [31]:
def register_source_func(x):
if x in ['AndroidApp', 'iOSApp']:
return 'Mobile app'
else:
return 'Non mobile app'
member['RegisterSourceTypeDef'].map(register_source_func).value_counts()
Out[31]:
Summarizing DataFrame with built-in methods¶
In [32]:
order_main['TotalSalesAmount'].sum()
Out[32]:
Summarizing and grouping DataFrame with built-in methods¶
In [33]:
order_main.groupby('PaymentType')['TotalSalesAmount'].sum()
Out[33]:
Once general operations are done, we can move on to advanced DataFrame operations¶
- Text to
datetimetype - Dealing with missing values
- Merging(joining) dataframes
Text to datetime type¶
In [34]:
print(behavior['HitDateTime'][:100])
In [35]:
print(pd.to_datetime(behavior['HitDateTime'][:100]))
In [36]:
print(member['RegisterDate'])
In [37]:
print(pd.to_datetime(member['RegisterDate']))
Dealing with missing values¶
- Using
isnullandnotnullto check ifnp.NaNexists - Using
dropnato drop rows withnp.NaN - Using
fillnato fillnp.NaNwith specific values
In [38]:
print(member.shape[0])
print("===")
print(member['RegisterSourceTypeDef'].isnull().sum())
print(member['RegisterSourceTypeDef'].notnull().sum())
print("===")
print(member.dropna().shape[0])
print("===")
print(member['RegisterSourceTypeDef'].fillna('Unknown').isnull().sum())
print(member['RegisterSourceTypeDef'].fillna('Unknown').notnull().sum())
Merging(joining) dataframes¶
mergefunction on columnjoinmethod on index
In [39]:
# Retrieve the order_main of max TotalSalesAmount
max_ttl_sales_amt = order_main['TotalSalesAmount'].max()
max_order_main = order_main[order_main.TotalSalesAmount == max_ttl_sales_amt]
max_order_main
Out[39]:
In [40]:
# Retrieve the order_slave of max TotalSalesAmount
max_transaction_num = max_order_main['TransactionNum'].values[0]
max_order_slave = order_slave[order_slave['TransactionNum'] == max_transaction_num]
max_order_slave
Out[40]:
We can use either UUID or TransactionNum as our foreign keys¶
In [41]:
pd.merge(max_order_slave, max_order_main, left_on='UUID', right_on='UUID', suffixes=('_slave', '_main'))
Out[41]:
If we want to adapt the join method, set FK to index first¶
In [42]:
max_order_slave = max_order_slave.set_index('UUID')
max_order_main = max_order_main.set_index('UUID')
max_order_slave.join(max_order_main, lsuffix='_slave', rsuffix='_main')
Out[42]:
Practices for pandas operations¶
- Import 4 91APP files into your own Python environment
- Fill
np.nanwithorder_slave['PaymentType']andorder_slave['ShippingType']for the maxTotalSalesAmountorder(TransactionNum: zK6XGNghLYS3ZuQloS4ocxEXeW8=)
Practices for pandas operations: Solutions¶
In [43]:
# Import 4 91APP files into your own Python environment
# Refer to previous slides
In [44]:
# Fill np.nan with order_slave['PaymentType'] and order_slave['ShippingType'] for the max TotalSalesAmount order(TransactionNum: zK6XGNghLYS3ZuQloS4ocxEXeW8=)
# Retrieve the order_main of max TotalSalesAmount
max_ttl_sales_amt = order_main['TotalSalesAmount'].max()
max_order_main = order_main[order_main.TotalSalesAmount == max_ttl_sales_amt]
# Retrieve the order_slave of max TotalSalesAmount
max_transaction_num = max_order_main['TransactionNum'].values[0]
max_order_slave = order_slave[order_slave['TransactionNum'] == max_transaction_num]
# Retrieve imputed values
impute_payment_type = max_order_main['PaymentType'].values[0]
impute_shipping_type = max_order_main['ShippingType'].values[0]
In [45]:
# Before fillna
print(max_order_slave['PaymentType'])
print(max_order_slave['ShippingType'])
In [46]:
# After fillna
print(max_order_slave['PaymentType'].fillna(impute_payment_type))
print(max_order_slave['ShippingType'].fillna(impute_shipping_type))
Data Visualization with Python¶
TL; DR¶
We briefly introduce visualization and how to use popular packages for visualization in Python with 91APP dataset.
What is data visualization?¶
Visualization is a tool that addresses the gap between data, function, formula, and viewers. Effective visualizations transform abstract collections of numbers into shapes that viewers quickly grasp and understand.
Why data visualization: using as a tool that addresses the gap between data and viewers¶
what are $x$ and $f$?
In [48]:
print(x)
print(f)
Gap addressed: data and viewers¶
$$f(x)=sin(x), \text{where } 0 \leq x \leq 4\pi$$In [50]:
plot_sinx(x, f)
Why data visualization: using as a tool that addresses the gap between function, formula and viewers¶
$$S(x) = \frac{1}{1 + e^{-x}}$$Gap addressed: function, formula and viewers¶
In [52]:
plot_sigmoid(x, S)
Visualization also plays a key role in machine learning¶
Machine learning engineer uses line plot to visualize cost function¶
In [54]:
plot_logistic_cost_function(x)
Machine learning engineer uses surface plot or contour plot to visualize gradients¶
Source: Coursera | Machine Learning
Machine learning engineer also uses contour plot to visualize gradients¶
Source: Coursera | Machine Learning
Machine learning engineer uses scatter plot and contour plot to visualize decision boundary¶
In [56]:
plot_multiple_decision_boundary()
The 2 most influential visualizations of ALL TIME¶
Minard's map of Napoleon's disastrous Russian campaign of 1812¶
In 1812, Napoleon marched to Moscow. 98% of his soldiers died. The simple but fascinating temperature line below the viz shows how cold ultimately defeated Napoleon’s army.
Hans Rosling's 200 Countries, 200 Years, 4 Minutes¶
A 2007 TED talk for the Swedish scientist shared his passion with the world. It shows the relationship between wealth and health.
Visualization plays heavy roles in¶
- Visualization
- Modeling
- Communicattion
Popular libraries we can use in Python for visualization¶
pandasseabornmatplotlibplotly
Difference between these libraries¶
- Plotting with
DataFrameorSeriespandasseaborn
- Plotting with
list,ndarray, orSeriesmatplotlib
- Plotting with interactivity
plotly
Importing these libraries¶
In [57]:
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
Getting data for a quickstart on these libraries¶
In [58]:
order_slave_dtypes = {
'PaymentType': 'str',
'ShippingType': 'str'
}
behavior_dtypes = {
'CategoryId': 'str',
'SearchKeyWord': 'str',
'TransactionNum': 'str'
}
member = pd.read_csv('MemberDataForNTU.csv')
order_main = pd.read_csv('OrderDataForNTU.csv')
order_slave = pd.read_csv('OrderSlaveDataForNTU.txt', dtype=order_slave_dtypes)
behavior = pd.read_csv('BehaviorDataForNTU.csv', dtype=behavior_dtypes)
Choose appropriate plots according to analytical purposes¶
- Bar for ranking and comparison
- Scatter for correlation
- Histogram for distribution
- Line for trend
- Box for distribution across different categories
How to plot with pandas¶
DataFrame class has a .plot() method with several key parameters.
Make a groupby summary with pandas¶
In [59]:
order_main.groupby('PaymentType')['TotalSalesAmount'].sum().sort_values()
Out[59]:
Make a barh with pandas following groupby summary¶
In [60]:
sale_amount_by_payment_type = order_main.groupby('PaymentType')['TotalSalesAmount'].sum().sort_values()
sale_amount_by_payment_type.plot(kind='barh')
plt.show()
Make another groupby summary with pandas¶
In [61]:
order_main['TradesDate'] = pd.to_datetime(order_main['TradesDate'])
order_main.groupby('TradesDate')['TotalSalesAmount'].sum()
Out[61]:
Make a line with pandas following groupby summary¶
In [62]:
sale_amount_by_trades_date = order_main.groupby('TradesDate')['TotalSalesAmount'].sum()
sale_amount_by_trades_date.plot(kind='line')
plt.show()
How to plot with seaborn¶
Using different plotting functions
barplotfunction for a barlineplotfunction for a line
Transform groupby summary to a DataFrame with pandas¶
In [63]:
sale_amount_by_payment_type_df = pd.DataFrame(sale_amount_by_payment_type).reset_index().sort_values('TotalSalesAmount', ascending=False)
sale_amount_by_payment_type_df
Out[63]:
barplot function for a bar¶
In [64]:
sns.barplot(x='TotalSalesAmount', y='PaymentType', data=sale_amount_by_payment_type_df)
plt.show()
Transform groupby summary to another DataFrame with pandas¶
In [65]:
sale_amount_by_trades_date_shipping_type = order_main.groupby(['TradesDate', 'ShippingType'])['TotalSalesAmount'].sum()
sale_amount_by_trades_date_shipping_type_df = pd.DataFrame(sale_amount_by_trades_date_shipping_type).reset_index()
sale_amount_by_trades_date_shipping_type_df
Out[65]:
lineplot function for a line¶
In [66]:
sns.lineplot(x='TradesDate', y='TotalSalesAmount', hue='ShippingType', data=sale_amount_by_trades_date_shipping_type_df)
plt.show()
How to plot with matplotlib¶
- Arranging data to iterables:
list,ndarrayorSeries - Call
figurefunction for a "figure" object - Call
axesfunction for an "axis" object - Call plotting methods from an axis object
- Add elements
- Call
showfunction for display
Take a bar for example: figure and axes first¶
In [67]:
fig = plt.figure()
axes = plt.axes()
Take a bar for example: call plotting methods and show plot¶
In [68]:
fig = plt.figure()
axes = plt.axes()
axes.barh(sale_amount_by_payment_type.index, sale_amount_by_payment_type.values)
plt.show()
Take a line for another example: figure and axes first¶
In [69]:
fig = plt.figure()
axes = plt.axes()
Take a line for another example: call plotting methods and show plot¶
In [70]:
fig = plt.figure()
axes = plt.axes()
x = sale_amount_by_trades_date.index
y = sale_amount_by_trades_date.values
axes.plot(x, y)
plt.show()
The line seems a bit too narrow for us...maybe it's time to add interactivity on it¶
In [71]:
import plotly.graph_objs as go
In [72]:
# Create df
df = pd.DataFrame(sale_amount_by_trades_date).reset_index()
# Create figure
layout = go.Layout(
autosize=False,
width=720,
height=480,
margin=go.layout.Margin(
l=50,
r=50,
b=100,
t=100,
pad=4
),
bargroupgap=0.3
)
fig = go.Figure(layout=layout)
In [73]:
fig.add_trace(
go.Scatter(x=list(df.TradesDate), y=list(df.TotalSalesAmount)))
# Set title
fig.update_layout(
title_text="Time series with range slider and selectors"
)
In [74]:
# Add range slider
fig.update_layout(
xaxis=dict(
rangeselector=dict(
buttons=list([
dict(count=1,
label="1m",
step="month",
stepmode="backward"),
dict(count=6,
label="6m",
step="month",
stepmode="backward"),
dict(count=1,
label="YTD",
step="year",
stepmode="todate"),
dict(count=1,
label="1y",
step="year",
stepmode="backward"),
dict(step="all")
])
),
rangeslider=dict(
visible=True
),
type="date"
)
)
In [75]:
fig.show()